1. Field of the Invention
The present invention is related to a picture coding apparatus and a picture coding method, for re-coding a digital compressed picture signal to obtain such a digital picture signal having either the same compression format or the different compression format from that of the digital compressed picture signal. More specifically, the present invention is directed to a picture coding apparatus and a picture coding method, capable of increasing a bit using efficiency when a digital compressed picture signal as an input picture signal is re-coded so as to reducing a number of frame skipping operations in the case that a total frame number of the digital compressed picture signal per unit time is varied.
2. Description of the Related Art
Conventionally, television conference systems using a narrow band ISDN have been practically utilized. Such a television conference system may be realized by a picture compression technique capable of compressing digitalized conference pictures so as to obtain a predetermined amount of conference data which can be transferred in real time within the frequency band of the narrowband ISDN. FIG. 8 represents an example of a picture transmission in a television conference system. In general, narrow bands (low bit rate) of either 64 kilobits per second (64 Kbps) or 128 Kbps, which is narrow in order to transfer pictures, are employed as a transmission path in such a television conference system.
In general, as a picture compression system for such a low bit rate, the ITU-T recommendations H.261 and H.263, and the specification 14496-2 of ISO/IEC (usually referred to as an “MPEG-4 Video”), and the like are used. While picture compression operations by way of these picture compression systems are utilized in real-time communication such as television conference system, the below-mentioned conditions should be satisfied:                Amounts of generated picture codes do not exceed a transmission band.        Delays are small, which do not disturb communications.        
To satisfy these conditions, in such a picture compression operation, two different rate controls called as a “quantizing control” and a “frame skipping control” are carried out, so that coding operations capable of satisfying these conditions are realized.
The quantizing control corresponds to such a control in which code amounts allocated to each picture and each macroblock contained in a picture are determined, and quantizing steps are determined based upon these determined code amounts. The frame skipping control corresponds to such a control in which when an amount of data which have already been coded but have not yet been transferred (namely, output buffer remaining amount) exceeds a constant amount, the next picture is not coded, but skipped.
As a concrete example of such a rate control, there is the method described in “MPEG-4 Video Verification Model version 16.0” of ISO/IEC JTC1/SC29/WG11 N3312 (will be abbreviated as a“VM16” hereinafter). This VM16 indicates an example of realizing the concrete compressing/decoding system of MPEG-4 Video (ISO/IEC 14496-2), and implies the general-purpose system having high practical possibilities. FIG. 9 shows a flow chart for describing the rate control executed based upon VM16.
At a step 10 of this flow chart, a parameter required for executing the rate control is set as an initializing process. At a step 20, a target code amount allocated to a picture which is presently encoded is calculated in accordance with the below-mentioned sequence (expressions are described by C language program. Various sorts of below-mentioned process operations are similarly described by C Language program).T=Max(Rs/30, Rr/Nr*0.95+S*0.05);T=T*(B+2*(Bs−B)/(2*B+(Bs−B));if(B+T>0.9*Bs)T=Max(Rs/30, 0.9*(Bs−B));else if(B−Rp+T<0.1*Bs)T=Rp−B+0.1*Bs;
It should be understood that:
Max(a, b): a larger value among “a” and “b”,
Min(a, b): a smaller value among “a” and “b”,
T: a target code amount,
Rs: a bit rate with respect to a certain image segment,
Rr: a code amount which is left so as to code remaining frames within a subject image segment,
Nr: a remaining frame number which should be coded within the subject image segment,
S: a code amount used when a preceding frame is coded,
B: a present remaining amount of an output buffer, and an initial amount is Bs/2,
Bs: an output buffer size,
Rp: a code amount extracted from the output buffer within 1 frame period (namely, inverse number of frame rate).
Subsequently, at a step 30, a quantizing value is determined in response to the target code amount acquired at the step 20. At a step 40, after the coding operation with respect to 1 frame is accomplished, an output buffer remaining amount, a remaining picture number, and the like are updated. Then, at steps 50-1 and 50-2, when the output buffer remaining amount exceeds 80% of the output buffer capacity, the next frame is skipped by the frame skipping control.
In this flow operation, an image segment indicates a group (cluster) of pictures which are theoretically segmented by certain constant time. For instance, assuming now that a segment is separated at every 1 second, a total number of pictures contained in this segment is equal to a frame rate. For example, in the case that pictures are coded at a frame rate of 10 pictures per 1 second, 10 screens of pictures are contained by 1 segment.
As described above, in order to execute the rate control of VM16, a coding frame rate is determined, and a coding operation should be carried out based upon the coding frame rate. Other general-purpose methods which are not described in this specification are also performed in a similar manner. That is, while coding frame rate is determined, a code amount to be allocated is calculated by using the determined coding frame rate.
In this case, a coding frame rate owns such an implication of a parameter which indicates a time duration defined from a time instant when a coding operation is presently carried out up to a coding operation of a next frame. For instance, when a coding frame rate is equal to 10 pictures/second, it can be seen that a time duration after one picture has been coded until a next picture is coded is equal to 0.1 second. Considering now of coding operations executed in the terminals of the television conference system exemplified in the example of FIG. 8, a coding frame rate may be freely set on the side of the transmission terminal in such a manner that this coding frame rate becomes equal to an integer-th of the output frame rate of the television camera, whereas images may be acquired under stable condition by the television camera in the compression unit provided on the side of the transmission terminal.
On the other hand, while terminals which employ different picture compression systems from each other are gradually popularized, there is such a need that pictures are transmitted/received among these terminals. In this case, apparatus for converting a picture compression system to another (will be simply referred to as “system converting apparatus” hereinafter) and vice versa are required between terminals. In FIG. 10, there is shown such a system for performing a picture communication between terminals having different sorts of picture compressing systems via a system converting apparatus. In a picture compressing unit 911 of a television conference terminal 91 provided on the transmission side, a rate control is carried out with respect to a picture signal which owns the known constant frame rate and is derived from a television camera. Then, the rate-controlled picture signal is transmitted via a transmission unit 912 to a transmission path, and is inputted into a system converting apparatus 92. As indicated in FIG. 11, a conversion unit 922 of the system converting apparatus 92 is arranged by an decompression unit 9221 and a compression unit 9222. The decompression unit 9221 decompresses a picture, compressed by a compression system of an input side and derived from a reception unit 921, so as to produce a decompressed picture. The compression unit 9222 recompresses this decompressed picture by way of a compression system of an output side. The recompressed picture signal is sent via a transmission path to a television conference terminal 93 provided on the reception side. The recompressed picture signal received by a reception unit 931 is converted by an decompression unit 932 into a decompressed picture which will be displayed on a television monitor.
However, in the system shown in FIG. 10 and FIG. 11, the input picture of the picture compression unit 911 of the television conference terminal 91 provided on the transmission side corresponds to the picture having the known constant frame rate, whereas the input picture of the compression unit 9222 employed in the system converting apparatus 91 corresponds to such a picture which has been once coded by the low bit rate and thereafter is decompressed. In the picture compression system of the low bit rate such as MPEG-4 video and the ITU-T recommendation H.261, since information of the frame rate is not contained in a stream, the frame rate of the input picture of the picture compression unit 9222 employed in the system converting apparatus 92 is unknown. Furthermore, there are many cases that frames are skipped in the frame skipping control during the coding operation in the terminal of the transmission side. In this case, the frame rate is not constant. In other words, after one picture is inputted, it is not clear when the next picture is inputted. As a result, in such a case that the above-explained general-purpose rate control is employed when the picture coding operation is carried out within the system converting apparatus 92, the below-mentioned problem will occur.
First, even when the coding frame rate is set to, for example, 15 pictures/second, in the case that the frame rate of the input frame is only 7.5 pictures/second, the pictures can be actually coded by only 7.5 pictures/second. However, since the frame rate employed in the calculation of the target coding amount in the rate control is equal to 15 pictures/second, the allocated code amount is small for the picture to be compressed at 7.5 pictures/second. The produced amount is decreased as compared with the allowed range (frame skipping threshold value). Then, in the case that the interval is furthermore prolonged since the input image itself is frame-skipped, the output buffer cannot afford to produce the codes. As a result, there is no data in the output buffer to be transmitted, the time duration during which the transmission path is used in a useless manner is prolonged.
FIG. 12 shows this example. FIG. 12 represents such an example of a transition of an output buffer remaining amount when the above-explained code amount allocating method by VM16 is employed. It should be noted that as to similar drawings, involving FIG. 12, which represent transitions of output buffer remaining amounts, abscissas thereof denote a time instant. Then, ordinates of these drawings show bit amounts left in the output buffer with respect to an output buffer remaining amount (line “B” shown in drawings) by values along a right axis. At the same time, with respect to a histogram of either an output picture or an input picture, the ordinate of this histogram represents any values other than the zero in the case that the relevant picture is present at the relevant time instant, and also shows the zero in the case that the relevant picture is not present at this relevant time instant, namely, shows as to whether or not the picture is present. As a consequence, a difference between a total number of non-zero values of an input picture and a total number of non-zero values of an output picture indicates a total number of pictures which are frame-skipped by the frame skipping control. Also, this graphic representation of FIG. 12 simulates operations of an encoder (coding device) accomplished by VM16 by setting the following assumption:
(1) a code amount which is actually produced by coding each of pictures is equal to an allocated code amount which is determined by the rate control; and
(2) an input picture is entered at 10 pictures per second, and each of the inputted pictures is coded only one time at maximum (namely, same picture is not coded plural times). Other setting values are given as follows:
A length of 1 image segment=1 second,
Rs: a bit rate of an image segment=64 kilobits,
a total picture number contained in 1 image segment=a coding frame rate,
Bs: An output buffer size=0.2 seconds of a frame=12800 bits,
Rp: an extracted code amount per 1 frame period=64 k/coding frame rate.
Originally, generally speaking, a code amount of each picture is theoretically simulated in such a manner that all of codes for one picture are immediately generated. However, it is so simulated that time is required for generating these codes, due to some reason. It should also be noted that this difference has no direct relationship as to a gist of the present invention. In this drawing, there are spaces between a second picture and a third picture, and between a fourth picture and a fifth picture, so that a frame rate is not constant. In the case that a coding frame rate larger than the actual coding frame rate is set in a fixed manner with respect to such an unstable frame rate, such large useless time of the transmission path occurs in the above-explained prior art, namely, there is completely no code to be transferred.
A further explanation is made of this reason. That is, in the example shown in FIG. 12, the coding frame rate is set to 30 pictures/second. As a result, while the code amount is allocated, such a code amount by which a picture transmission is completed by 1/30 seconds may constitute a reference of the code amount allocated to each of the pictures. This is a target code amount before being corrected, and this target code amount is approximately 2.1 kilobits (=2133 bits) in this example. On the other hand, since the coding frame rate is equal to 10 pictures/second even at such a place that no frame is skipped in the actual input picture, the interval between the pictures is equal to 1/10 seconds. In a general-purpose rate control system involving VM16, since an actual output buffer remaining amount is small, the target code amount is corrected in such a manner that the allocation code amount is made larger than the reference code amount. However, the correction corresponds to a coefficient with respect to the reference code amount, and becomes approximately 2 in maximum in VM16. This correction may have an effect in view of the following implication. That is, in the case that pictures can be entered under stable condition which is set by the coding unit, the code amount may be stably allocated. However, as explained in this example, in the case that the input picture owns the different coding frame rate from the set coding frame rate, there is a drawback that the difference between the set coding frame rate and the actual coding frame rate cannot be sufficiently solved. In this example, the target code amount which is corrected based upon the coefficient becomes approximately 4300 bits, so that such a condition is continued under which the output buffer remaining amount becomes low.
Furthermore, in the case that an input picture is dropped under this condition, since there is no spare code in the output buffer remaining amount, there is a lack of codes to be transferred. This implies that the code amount by which the codes could be originally transferred is not completely allocated. In general, the larger the code amount is allocated, the higher the image quality can be improved in the same coding system. As a result, in this case, such an image quality which could be originally achieved would be deteriorated. As previously explained, in the conventional rate control system by the coding operation, this rate control system is established based upon such an initial condition that such a picture adapted to the set coding frame rate is entered. As a consequence, as explained in the code converting apparatus shown in FIG. 12, when the picture adapted to the set coding frame rate cannot be inputted, the effective code amount allocation cannot be carried out.
Secondly, there is another problem that when the coding frame rate is set to a large coding frame rate so as to solve the above-explained problem, frame skipping operations occur many times. This example is shown in FIG. 13. FIG. 13 shows an output buffer remaining amount in the case that while a coding frame rate is set to 7.5 pictures/second, a coding operation is carried out with respect to an input picture whose coding frame rate is 10 pictures/second. As apparent from the drawing, although 11 pictures are entered, only 5 pictures are outputted, and thus, the remaining 6 pictures are frame-skipped. Among these six pictures, skipping of the two pictures is caused by the mismatching of the set coding frame rate with respect to the input frame rate, which cannot be solved by the rate control system. However, skipping of the remaining four pictures is caused by the frame skipping control.
This reason will now be explained. As previously described, in the conventional rate control method, there is such an initial condition that the pictures can be coded in accordance with the set coding frame rate. As a result, if the set coding rate is selected to be 7.5 pictures/second, then such a code amount that a transmission of one picture is completed within 1/7.5 seconds may constitute the reference of the allocation code amount. In the example shown in FIG. 13, since this set coding frame rate is mismatched with the actual input frame rate, the output buffer remaining amount is not stable, but also, the correction value may cause the target code amount to be excessively increased, or decreased. Accordingly, the frame skipping operations are usually carried out many times. In this case, the (motion) picture to be outputted may become (motion) pictures lacking smoothness, as compared with the input (motion) pictures. Moreover, the code amount with respect to such a picture which is originally to be coded becomes useless, and the image quality would be deteriorated.
As previously described, when the conventional rate control system is carried out in the system converting apparatus, the following problems may occur: That is, (1) since a frame rate of an input picture is unknown and variable, an appropriate target code amount cannot be set, but also such a code amount which can be originally used cannot be effectively used, so that the image quality is deteriorated. (2) Since frame-skipped motion pictures, which dropped frames prior to the input (to this system conversion apparatus), is inputted, a large number of necessary frames are skipped while the frame skipping operation is carried out during the conversion operation.